#1)

Add an admin-only way to generate the number to pin: CanonicalVerifier
.measure_baseline(replicates) admin-scores the fixed seed N times on the
selection partition and each target and returns per-key mean/stddev. Exposed
as POST /score/baseline and 'vero harbor score-baseline --replicates N'.
Reuses the existing admin scoring machinery.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…atten

The benchmark-config tests read harness-engineering-bench/ (a separate branch
in the stacked split), so skip them when it isn't checked out. Update the
paths for the candidates/ -> top-level flatten (all benchmarks are now
top-level siblings of gaia/).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- optimizer.run: validate self.max_proposals >= 0 before the proposal_limit
  guard, so the specific 'must be non-negative' error is reachable (was a
  dead branch that always tripped the generic message first).
- compiler._safe_extract_tar: extractall(filter='data') to strip device
  files / setuid bits and neutralize unsafe links, matching
  extract_harbor_session_archive (also silences the py3.14 deprecation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Update harbor_requirement pin from harbor[modal]==0.18.0 to ==0.20.0
across the README and harbor tests. Remove the PYTHONPATH injection in
the harness eval launch: it was a no-op (uv strips PYTHONPATH from the
child env) and did not fix the "No module named <agent>" failure under
the dropped uid, so it and its test assertion are gone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The candidate is checked out at <mktemp>/repository; `mktemp -d` makes
that parent 0700 root. The eval launch chowned the repository (and the
staging tree) to the harness user but not the mktemp parent, so once the
process dropped to harness it could not traverse into its own workspace
to resolve the editable candidate package's absolute path — every eval
(baseline included) failed with "No module named <agent>" while harbor
itself still loaded from the harness-owned cache.

Grant the harness user traversal on that parent (chmod o+x) alongside the
existing chowns. o+x suffices because run_as drops to the user's own
uid+gid, so it is "other" relative to the root-owned parent. The dir
holds only candidate code, so widening traversal leaks no trusted data.

Reproduced and fixed against the real compiled sidecar image with a
faithful user+group+extra_groups privilege drop: 0700 -> "No module
named gaia_agent"; 0701 -> import resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add a reachability probe: after chowning/granting traversal, run `test -r
<project_path>` as the dropped user before launching harbor. Readability
requires traversing every ancestor, so any provisioning gap (a missing
chown, a non-traversable parent) is caught here with a clear message
instead of resurfacing as a cryptic "No module named <agent>" several
retries downstream — which is exactly how the checkout-parent bug hid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Extract the harness provisioning commands and reachability probe into
vero.harbor.isolation as a single source of truth, and have the eval
launch use them, so the runtime and the test can't drift.

Add tests/test_v05_harbor_isolation_container.py: inside a throwaway
Linux container it loads the real sandbox.py + isolation.py standalone
(stdlib-only, no install) and asserts, against the real uid/gid drop and
the real provisioning commands, that (1) a checkout under a 0700 mktemp
parent is unreachable to the dropped user when only the leaf is chowned
(the regression sentinel for the shipped bug), (2) it becomes reachable
after harness_grant_commands and the probe agrees, and (3) trusted
root-only state stays unreadable to the dropped user. Skips when no
docker daemon is available.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

One well-named entry point over the .evals/ context — the new avatar of the
legacy VeroAgent's three structured tools. Runner subcommands (run/status/
result/submit) delegate to the sidecar client lazily; the viewers (list/show/
cases/trace/diff/plan/tasks) are unprivileged stdlib readers of the disclosure-
projected tree, with bounded, paginated, metadata-first output. The compiler
bakes skills/evals/SKILL.md into the optimizer workspace by default
(include_evals_skill), and the instruction template now teaches `evals`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

min_aggregate_cases moves onto the core EvaluationAccessPolicy (omitted
resolves to 5 under aggregate disclosure, 1 otherwise) and the engine refuses
agent-chosen aggregate subsets below the floor after cost resolution, so the
protection holds on every path — programmatic API and local runs included —
not just compiled harbor deployments. The sidecar keeps its earlier check as
defense-in-depth and now passes the floor through its explicit authorization.

AgentAccessSpec.to_access_policy() is the single typed translation from build
spec to runtime policy; the compiler emits it nested in serve.json and
SidecarEvaluationPolicy holds it as `access`, replacing the hand-rolled flat
keys (disclosure/agent_evaluable/min_aggregate_cases/expose_case_resources)
that had to be kept in sync across three representations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Its evaluations were running entirely off-gateway. The live run shows 8
completed evaluation jobs, 0 requests in the gateway's evaluation scope, and
case metrics carrying only `agent_reported_*` tokens -- 41.3M input against an
allow-list that was never consulted.

Cause is one missing lookup. browsecomp-plus sets
`task_services_use_upstream`, which exists so an in-container grader can reach
the provider and does that by pointing OPENAI_* at the real upstream; the
candidate agent's metered credential then arrives only on
VERO_AGENT_INFERENCE_*. tau3 and swe-atlas-qna read those. browsecomp-plus
did not, so it fell through to OPENAI_* and ran on the raw upstream
credential: unmetered, unbudgeted, and with the pinned target model
unenforced.

Checked the consequence rather than assuming it: the shipped candidate from
the +0.169 run touches only search strategy -- turn cap, snippet size, dedup
-- and never the model or credentials, so that result stands as a harness
improvement. The capability to change the model was simply unguarded, which is
luck rather than a property, and 10 contestants x 3 repeats is a lot more
chances to stumble into it.

The existing config-level test could not catch this: the build does declare a
gateway and vero injects the upstream on purpose, so it passed throughout. The
new test asserts against the agent source instead, skips benchmarks that do not
reroute, and imports the env var names from vero.harbor.backend so it cannot
drift from the contract. Verified it fails with the agent fix reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A five-agent review of the suite turned up two documented safety guarantees
that the enforcing code contradicts. Both are in the files a colleague reads
first, and a false assurance is worse than no assurance.

CONFIGURATION.md said the evaluation allow-list fixes the target model "so a
candidate cannot swap it". gateway/inference.py carries a note, at the exact
line that enforces it, saying this is explicitly not a hard guarantee: both
scopes share one gateway host split only by URL path, and the optimizer holds
the producer token and authors the candidate. The doc now states the real
property and points at that note.

SKILL.md said "The optimizer never sees the real upstream key. Keep it that
way." Three of the five promoted benchmarks set task_services_use_upstream,
which puts the raw upstream credential into the evaluation sub-run, and those
three also run the candidate harness without a separate user -- an exposure
swe-atlas-qna's own build.yaml labels "ACCEPTED EXPOSURE". The skill now names
both caveats up front rather than hedging one of them 200 lines later.

Three factual corrections a reader would otherwise act on: the pinned
baseline_reward is inert at runtime (verified against a live run's
finalize.json, which carries an empty baseline_rewards, because the verifier
reads the pin only inside its score_baseline branch); max_concurrency is
concurrent trials, not cases, so 24 is 8 cases in flight once the test target
runs 3 attempts; and the token caps are 4.4-6.8M per case-run, 3.3-5.1x the
worst measured cost, not the stated ~3M and 2.3x.

Also corrected a claim I introduced earlier today: an omitted finalization
scope does not let search starve held-out scoring. The compiler mints that
token unconditionally and the gateway keys each ledger by scope, so an unset
scope gets its own pool at evaluation's size. The starvation incident predates
the reserved scope.

Finally, swe-bench-pro sits at the top level, which this README defines as
meaning promoted, while being absent from the promoted table and from every
column of CONFIGURATION.md. It is now explicitly marked unpromoted, with the
specific conventions it misses, because launching it yields a number that is
not a measurement of the same quantity.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The library README now leads with a run that actually happened, through the
contained path rather than the local one: mini-swe-agent with sonnet-5 on
harbor-circle-packing, 0.9598 to 2.5766 on the held-out partition, shipped
true, from a 15 KB Lubachevsky-Stillinger growth algorithm with LP refinement.
The figure is generated from the run's own session by a stdlib-only script, so
it regenerates rather than being a drawing.

It also records what the same task did without a prohibition on hardcoding:
codex scored 2.6360 in eight minutes by copying the published Packomania table
into 26 coordinate literals. Higher than the honest run, and held-out scoring
cannot tell the difference, because every partition holds one deterministic
case so a memorized answer transfers perfectly. instruction.md now forbids it.
That contrast is the more useful thing to put in front of a reader than either
number alone.

One detail in the figure earns its place: an infeasible candidate, rejected on
the `valid == 1` constraint, immediately followed by the agent's own "Add
safety margin to guarantee strict feasibility" commit. The constraint did work.

The verifier now reports a pinned baseline_reward whether or not
`score_baseline` is set. That flag decides whether the seed is *scored*, which
is expensive; it should not decide whether the run states the number every
delta is measured against. Read only inside the score_baseline branch, the pin
left every `score_baseline: false` run -- which is all five promoted
benchmarks -- shipping an empty `baseline_rewards`, confirmed against a live
officeqa finalization. Test reverted to check it fails without the fix.

Also here: the native optimizer falls back to OPENAI_* when LITELLM_* is
unset, since only reading the latter meant a standard environment posted to a
route the proxy does not serve and returned a 403 that reads as an auth error;
the session mismatch error now names the differing field and the remedy,
because a changed optimizer.model is enough to trigger it; the root README
answers "where is the paper code" once instead of twice; both mermaid diagrams
are replaced with one ASCII figure of the actual pieces; and
harbor-circle-packing documents the two prerequisites that cost three failed
starts today -- the required -p, and vendoring vero into the build context.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The note added earlier today said the pin was inert at runtime, which was true
when written: the verifier read `target.baseline_reward` only inside its
`score_baseline` branch, so all five promoted benchmarks shipped an empty
`baseline_rewards`. pr2 fixed that, so the doc now says the pin is reported in
`finalize.json` and flags that runs before 2026-07-28 carry an empty field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Greptile (5/5, safe to merge) left this open from an earlier round. Candidate
selection tries the durable submission and then falls through to `_auto_best`
unconditionally, but only `auto_best` mode is validated to carry backend_id,
evaluation_set and objective. So a `submit`-mode run whose agent never wrote a
submission reached three bare asserts: an AssertionError with no diagnostic,
and under -O no check at all. It now declines and drops to `_pick_last`, the
intended last resort. Test reverts the guard to confirm it reproduces the
AssertionError.

Not fixed, deliberately: the same review flagged the unshielded status write in
_execute_tracked_job's Exception handler. Shielding it after `admitted.set()`
makes the common path worse -- shield schedules a task and therefore yields, so
the caller observes the job still QUEUED, which broke the admission-denial
test. Shielding it before `admitted.set()` deadlocks, so that ordering is
load-bearing. The review rates the consequence as a misleading status under a
narrow double-cancellation race that self-corrects on reload, which is cheaper
than either regression.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Greptile (5/5, safe to merge) named the two gaps on this PR.

UPSTREAM_MODEL_UNAVAILABLE had no tests. It is the taxonomy's most dangerous
silent failure -- unclassified, a model that is configured but not deployed
makes every call 404, the agent writes no answer, and each case is recorded as
an informative task failure, i.e. the harness scoring the candidate down for a
model that does not exist. Tests now pin the provider error codes it matches,
its terminating/not-a-sample/counts-toward-invalidity policy, and the
deliberate narrowness: a bare "does not exist" must stay unmatched, or it would
swallow "loading container: file does not exist".

load_build was annotated `-> dict` while returning `(dict, Path)`; the caller
unpacks two values. Corrected rather than merely added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A missing rate-limit header reads as 0, and the label compared the two derived
capacities directly, so whichever axis was absent won: with no TPM header and
an RPM limit, `0 <= by_rpm` printed "TPM-bound" and would send someone to raise
a quota that was never the constraint.

`fits` already skipped zeros; only the label did not. Report the axis that
produced `fits`, and say so explicitly when the other header is missing rather
than implying both were read.

Greptile flagged this and proposed `by_tpm and by_tpm <= by_rpm`, which fixes
the absent-TPM case but still prints "RPM-bound" for the mirror case (a TPM
limit and no RPM header), because the `and` short-circuits to the same branch.